Training Feedforward Neural Networks Using Orthogonal Iteration of the Hessian Eigenvectors
نویسنده
چکیده
Training algorithms for Multilayer Perceptions optimize the set of Wweights and biases, w, so as to minimize au error t%nction,E, applied to a set of N training patterns. The well-known back propagation algorithm combines an efficient method of estimating the gradient of the error function in weight space, AE=g, with a simple gradient descent procedure to adjust the weighb, Aw = –qg. More efficient algorithms maintain the gradient estimation procedure, but replace the update step with a faster non-linear optimization strategy [1].
منابع مشابه
A Comparison of First and Second Order Training Algorithms for Artificial Neural Networks
Minimization methods for training feed-forward networks with Backpropagation are compared. Feedforward network training is a special case of functional minimization, where no explicit model of the data is assumed. Therefore due to the high dimensionality of the data, linearization of the training problem through use of orthogonal basis functions is not desirable. The focus is functional minimiz...
متن کاملA Smooth Optimisation Perspective on Training Feedforward Neural Networks
We present a smooth optimisation perspective on training multilayer Feedforward Neural Networks (FNNs) in the supervised learning setting. By characterising the critical point conditions of an FNN based optimisation problem, we identify the conditions to eliminate local optima of the cost function. By studying the Hessian structure of the cost function at the global minima, we develop an approx...
متن کاملMemory Capacity of a Novel Optical Neural Net Architecture
A new associative memory neural network which can be constructed using optical matched filters is described. It has three layers, the centre one being iterative with its weights set prior to training. The other two layers are feedforward nets and the weights are set during training. The best choice of central layer weights, or in optical terms, of pairs of images associated in a hologram is con...
متن کاملA Spectral Version of Perry’s Conjugate Gradient Method for Neural Network Training
In this work, an efficient training algorithm for feedforward neural networks is presented. It is based on a scaled version of the conjugate gradient method suggested by Perry, which employs the spectral steplength of Barzilai and Borwein that contains second order information without estimating the Hessian matrix. The learning rate is automatically adapted at each epoch, using the conjugate gr...
متن کاملTowards a Mathematical Understanding of the Difficulty in Learning with Feedforward Neural Networks
Despite the recent success of deep neural networks in various applications, designing and training deep neural networks is still among the greatest challenges in the field. In this work, we address the challenge of designing and training feedforward Multilayer Perceptrons (MLPs) from a smooth optimisation perspective. By characterising the critical point conditions of an MLP based loss function...
متن کامل